Method for objects segmentation in video sequences by object tracking and user assistance

ABSTRACT

A video object segmentation method is disclosed in the present invention. This video object segmentation method is for correctly segmenting video objects. In the video object segmentation method, the video objects to be firstly appeared in a video sequence are segmented in manual or semi-manual, and then the consecutive video frames is automatically segmented by an object-tracking based on the moving of the primarily segmented video object. In other words, the video object segmentation method applicable to a video system, comprises a first step of primarily segmenting objects existing in a frame of video sequence manually or semi-manually; and a second step of automatically segmenting the objects within the consecutive frames. This video object segmentation method can be adapted to a video system.

CROSS REFERENCED TO RELATED APPLICATION

[0001] This application is a continuation of U.S. patent applicationSer. No. 09/701,822 filed Jan. 26, 2001

TECHNICAL FIELD

[0002] The present invention relates to a video object segmentationmethod for correctly segmenting a video into a plurality of videoobjects by a user's assistance and an object tracking, when a videoobject-oriented manipulation, an editing, an object-based userinteraction, an object-based video coding and the like are performed inan apparatus such as a video editing, creating and compositingapparatus, and an object-based video codec (or encoder/decoder).

BACKGROUND ART

[0003] According to conventional video object segmentation methods,objects in image are automatically segmented based on the determinationwhether the brightness values of consecutive images are different ornot, or are manipulatively segmented for every frame by user using anediting tool.

[0004] In the conventional video object segmentation methods, theautomatic segmentation method causes the problem in that thesegmentation capability is varied in accordance with the threshold valueon which it is determined that the brightness value between theconsecutive images is changed, and the manipulative segmentation methodhas the problem in that it takes a considerably long time because thesegmentation of every frame requires the user's direct manipulationwork, thus it endows the user with the economical burden.

[0005] Accordingly, the present invention is devised for solve the aboveproblems. The objective of the present invention is to provide a videoobject segmentation method for correctly segmenting a video object, inwhich the video objects to be firstly appeared in an image sequence aresegmented in manual or in semi-manual, and then the following videoframes are automatically segmented by an object-tracking based on themoving of the primarily segmented video object, for the sake ofdetecting an object from an image in a video system.

DISCLOSURE OF INVENTION

[0006] To accomplish the objective of the present invention, there isprovided a video object segmentation method applicable to a videosystem, comprising the steps of: primarily segmenting an object existingin a frame of video sequence manually or semi-manually; and secondlysegmenting the object within a video sequence including the primarilysegmented object automatically.

[0007] In one preferred embodiment of the present invention, the videoobject segmentation method further includes the steps of: determiningwhether any scene change is made between consecutive frames or any newobject other than the primarily segmented object appears within theframe being automatically segmented, when repeatedly performing the stepof secondly segmenting for consecutive frames; and repeating the step ofprimarily segmenting, if the answer of the step of determining ispositive. This method also includes the steps of examining the qualityof automatically segmented results,; repeating the step of secondlysegmenting, if the quality of automatically segmented results issatisfactory; and repeating the step of primarily segmenting, if thequality of automatically segmented results is not satisfactory.

[0008] In addition, the step of primarily segmenting can be made bysegmenting the object within the image in completely manual using anuser interface tool, or by segmenting the object within the image insemi-manual, such that, if the user designates manually a rough boundaryline of the object within the image, then the object within the image isautomatically segmented based on the designation-related information andan image segmentation information. The step of secondly segmenting canbe made by tracking the object region to which in the current frame theprimarily segmented video object in the previous frame is moved, so asto segment the object within the frame of the consecutive frames.

BRIEF DESCRIPTION OF DRAWINGS

[0009] For a more complete understanding of the present invention andthe advantage thereof, reference is now made to the followingdescription taken in conjunction with the accompanying drawings inwhich:

[0010]FIGS. 1a and 1 b are diagrams of an exemplary video codec to whichthe method of the present invention may be applied; and

[0011]FIG. 2 is a flowchart for illustrating a preferred embodiment of avideo object segmentation method according to the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] The preferred embodiment of the present invention willhereinafter be described in detail with reference to the accompanyingdrawings.

[0013]FIGS. 1a and 1 b are exemplary diagrams of a video codec to whichthe method of the present invention can be applied.

[0014] Referring to FIG. 1a, the video encoding part to which the methodof this invention is applied, includes a video object segmentation unit101 for segmentation an externally input video sequences into videoobjects, a plurality of video object encoding units 102 forvideo-object-oriented coding the video objects inputted from the videoobject segmentation unit 101 according to the MPEG-4 (Moving PictureExpert Group-4) video coding algorithm so as for reducing the videoobject data, and a multiplexing unit 103 for multiplexing the reducedvideo object data inputted from the plurality of video object codingunit 102 according to the MPEG-4 multiplexing standard and thentransmitting/storing the multiplexed bit stream.

[0015] Referring to FIG. 1b, the video decoding part to which the methodof this invention is applied, includes a de-multiplexing unit 104 forde-multiplexing the transmitted/reproduced bit stream data, a pluralityof video object decoding unit 105 for decoding the de-multiplexed videoobjects inputted from the de-multiplexing unit 104, a picturecompositing unit 106 for reconstructing a picture based on the decodedvideo objects inputted from the plurality of video object decoding unit105, and a display unit 107 for displaying the reconstructed picture ofthe picture compositing unit 106 to the display unit.

[0016]FIG. 2 is a flowchart of a video object segmentation methodaccording to one embodiment of the present invention.

[0017] First, when the video intended to be segmented is input, thevideo objects appeared in the first frame of the video sequence or thenewly appeared video objects in the consecutive frames, are manuallysegmented by user or semi-manually segmented by user's manipulationbased on the spatial information such as luminance, color or the like(see step 201).

[0018] For example it is impossible that an exact mathematical modelingfor the video object on video sequences can be made and the measure forsegmenting the video objects can be defined. Thus, the user segments inmanual the video objects shown in the first frame or the newly appearedvideo objects in the consecutive frames by using a simple user interfacetool such as a mouse, or segments semi-manually them by user'smanipulation based on spatial information. More specifically, if theuser simply designates in manual a rough boundary line of the objectwithin the frame, then the object within the frame is automaticallysegmented based on the designation-related information and an imagesegmentation information.

[0019] Therefore, the video object is defined and segmented by theuser's manipulation. if it is one of the first frame of the videosequence, the frame on which the new object is revealed, or the frame onwhich a scene change is made from the previous frame,

[0020] Second, the video object is automatically segmented by performingobject-tracking based on the moving of the video object which has beendefined and segmented in the previous frame (step 202).

[0021] For example, an estimation is made for the object region to whichthe video object having been segmented in the previous frame is movedand the video object on the previous segmented mask is projected intothe segmented mask of the present frame in accordance with the movedamount. Here, an error and noise of the segmentation mask generated dueto the projection is compensated and corrected, and then the pixelhaving an segmentation labeling uncertain due to the projection, for thesegmentation mask, is segmented into an uncertain area. Then using thespatial information such as the luminance and color information of thecurrent frame, the pixel of the uncertain area is segmented into thenearest video object. Here, a median filter is used for the segmentationmask having been last segmented. This is for the sake that the outlineof the video object is streamlined so as to give a visually good effect.

[0022] Next, in step 203, it is analyzed whether a scene change is madeor a newly appeared video object is in the next frame. That is, beforesegmentation of the video object, it is detected whether a scene changeoccurs in the consecutive frames or any newly appeared video objectother than the segmented video objects exists.

[0023] If the result of the analysis shows that any scene change is madeor any new object exists in the frame, the objects which are newlyappeared or exist in the changed scene are segmented by manually orsemi-manually by user's assistance (step 201).

[0024] If the result of the analysis shows that there is no scene changeand any new object does not exist in the frame, in step 204 it isdetermined whether the automatic segmentation result by object-trackingfor the previous frame is satisfactory. This is that, since thesegmentation by object-tracking is performed using the video objectsegmented in previous frame, the erroneous segmentation for the previousresult cumulatively affects the following segmentation byobject-tracking to become incorrect. Here, the satisfaction examinationfor the segmentation result can be manually performed based on theuser's determination or can be automatically performed.

[0025] When the segmentation result by object-tracking is“unacceptable”, in step 201 the segmentation process by user'sassistance. Otherwise, in step 202 the segmentation process byobject-tracking for the next frame is performed.

[0026] According to one preferred embodiment of the present invention,the video object segmentation method segments the video object from thevideo sequences using various information such as a user's assistance,spatial information, temporal information of object-tracking and thelike, so as to reduce the time required for segmentation and the user'sendeavor and improve the correctness of the automatic video objectsegmentation.

[0027] As described above, according to the present invention, thedesired object can be effectively segmented from the video sequencesusing an user's assistance, spatial information, temporal informationrelated to object-tracking or the like. Also the video objectintractable to be defined or segmented is manually or semi-manuallysegmented by user's assistance, and thus-segmented video object isautomatically segmented by object-tracking. In other words, theeffective combination of the automatic segmentation and the manualsegmentation methods makes it possibly that the video objects arecorrectly segmented and the user's endeavor and time can be reduced.Thus, the method of the present invention has an effect that it can beapplied to a video editing, creating and compositing apparatus, anobject-based video codec or the like.

[0028] Although preferred embodiments of the present invention has beenillustrated and described, various alternatives, modifications andequivalents may be used. Therefore, the foregoing description should notbe taken as limiting the scope of the present invention which is definedby the appended claims.

What is claimed is:
 1. A video object segmentation method applicable toa video system, comprising the steps of: a) primarily segmenting objectsexisting in a frame of a video sequence manually or semi-manually; andb) automatically segmenting the objects within a video sequenceincluding the primarily segmented object.
 2. The video objectsegmentation method in accordance with claim 1, further comprising thesteps of: c) determining whether any scene change is made betweenconsecutive frames or any new object other than the primarily segmentedobject appears within the video sequence being automatically segmented,when repeatedly performing the step b) for consecutive frames; and d)repeatedly performing the first step, if the answer of the step ofdetermining is positive.
 3. The video object segmentation method inaccordance with claim 2, further comprising the steps of: e) examiningthe quality of automatically segmented results, if there is no scenechange between consecutive frames and any new object other than theprimarily segmented object does not appear within the video sequencebeing automatically segmented; f) performing the second step, if thequality of automatically segmented results is satisfactory; and g)repeatedly performing the first step, if the quality of automaticallysegmented results is not satisfactory.
 4. The video object segmentationmethod in accordance with claim 1, wherein the first step of primarilysegmentation is made by segmenting the objects within the frame incompletely manual using an user interface tool.
 5. The video objectsegmentation method in accordance with claim 1, wherein the first stepof primarily segmentation is made by segmenting the object within theframe in semi-manual, such that, if the user designates manually a roughboundary line of the object within the frame, then the object within theframe is automatically segmented based on the designation-relatedinformation and an image segmentation information.
 6. The video objectsegmentation method in accordance with claim 1, wherein the second stepof automatically segmentation comprises the step of: tracking the objectregion in the current frame to which the primarily segmented videoobject in the previous frame is moved, so as to segment the objectwithin the frame of the consecutive frames.
 7. The video objectsegmentation method in accordance with claim 5, wherein the imagesegmentation information is a spatial information including a brightnessinformation and a color information.